Electronically controlled graduated density filters in stacked image sensors

ABSTRACT

A method for performing color density filtering of images captured in a digital camera having a mechanical shutter and an imaging array including a plurality of pixels each including different color sensors aligned with each other, including opening the mechanical shutter, resetting all of the color sensors in each pixel by asserting a row reset signal, separately asserting color-select signals for each color after the mechanical shutter is opened, independently starting exposure for each different color sensor at a color sensor exposure start time by de-asserting its color select signal, the exposure start time for each different color sensor being a function of a color density filter function, closing the mechanical shutter, and reading color exposure values from the color sensors by separately asserting color-select signals after the mechanical shutter has closed, the reading being unrelated to the start times for the color sensors.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a division of U.S. patent application Ser. No.15/195,344, filed on Jun. 28, 2016.

BACKGROUND 1. Field of the Invention

The present invention relates to digital imaging techniques and todigital image capture techniques. More particularly, the presentinvention relates to electronic density filters and more particularly toindependent color channel graduated density filter techniques in digitalcameras.

2. The Prior Art

In landscape photography, the dynamic range in the scene often exceedsthe capability of image sensors in digital cameras. This is oftenbecause the upper portion of the frame includes the sky, which issubstantially brighter than the remainder of the subject matter in theframe. This problem also exists with print film.

It is very common therefore to use an optical neutral density filter tocompress the dynamic range of a scene to be within the dynamic range ofthe image sensor. As illustrated in FIG. 1, the density of the filter isgraduated and increases from the bottom of the image to the top of theimage.

The optical neutral density filter is placed in front of the lens, andis oriented such that the portion of the filter having the highestdensity is located at the top of the frame so that the image is darkenedrelative to the bottom. While the density change from the top of theimage to the bottom of the image can be linear as shown at referencenumeral 10, it is most often non-linear as shown at reference numeral12.

It is known that an optical neutral density filter can be electronicallysimulated by placing a variable charge sharing capacitance in parallelwith individual pixel sensors in an image sensor or by using a variablecapacitance to share the sensed charge of one or more pixel sensorphotodiodes during readout.

An example of such an electronic neutral density filter for an imagesensor is disclosed in U.S. Pat. No. 7,635,833 to Mansoorian. Inoperation, each photodiode is reset by turning on the appropriatetransfer gates, such as 310, at the same time as the reset transistor325. The photodiode is then allowed to integrate charge.

Because the accumulated photocharge in each pixel sensor is sharedbetween the photodiode and the added capacitor, the pixel sensoroperates as though the photodiode received fewer illumination photons byan amount proportional to the value of the capacitor. As the capacitanceincreases, the size of the photodiode effectively increases, and thesensitivity of the photodiode to incoming illumination decreases.

It is also known to provide an electronic neutral density filter byintroducing a controlled delay between the reset signal and the readsignal of a digital imaging array. Such a configuration is disclosed inU.S. Pat. No. 8,780,241 to Johnson.

BRIEF DESCRIPTION

According to one aspect of the present invention, an electronic neutraldensity filter is implemented in a digital camera using a mechanicalshutter and having an imaging array including a plurality of pixelsensors that each include a photodiode coupled to a floating node by apixel select transistor, a reset transistor coupled to the floatingnode, and a readout transistor coupled between the floating node and acolumn line by a row-select transistor. A method for performing neutraldensity filtering includes opening the mechanical shutter, turning onall of the reset transistors, for each row in the array, turning on allof the pixel select transistors, simultaneously turning off all of thepixel select transistors after an interval of time has expired afterturning on all of the pixel select transistors, turning off the resettransistors in the array after the interval of time has expired, closingthe mechanical shutter, wherein the interval of time for successive rowsdecreases as a monotonic function. The accumulated photocharge may thenbe read from the pixel sensors at a time after but otherwise unrelatedto the resetting of the pixel sensors.

According to another aspect of the present invention, an electronicneutral density filter is implemented in a digital camera using amechanical shutter and having an imaging array including a plurality ofpixel sensors that each include a photodiode coupled to a floating nodeby a pixel select transistor, a reset transistor coupled to the floatingnode, and a readout transistor coupled between the floating node and acolumn line by a row-select transistor. A method for performing neutraldensity filtering includes opening the mechanical shutter, turning onall of the reset transistors, for each row in the array, turning on allof the pixel select transistors, simultaneously turning off all of thepixel select transistors after an interval of time has expired afterturning on all of the pixel select transistors, turning off the resettransistors in the array after the interval of time has expired, closingthe mechanical shutter, wherein the interval of time for successive rowsis a non-monotonic function. The accumulated photocharge may then beread from the pixel sensors at a time after but otherwise unrelated tothe resetting of the pixel sensors.

According to another aspect of the present invention, a color electronicneutral density filter is implemented in a digital camera using amechanical shutter and having an imaging array including a plurality ofpixel sensors for different colors that each include a photodiodecoupled to a floating node by a pixel select transistor, a resettransistor coupled to the floating node, and a readout transistorcoupled between the floating node and a column line by a row-selecttransistor. A method is disclosed for performing neutral densityfiltering independently for each of the colors includes opening themechanical shutter, turning on all of the reset transistors, for eachrow in the array, turning on all of the pixel select transistors,simultaneously turning off all of the pixel select transistors for eachselected color separately after an interval of time has expired afterturning on all of the pixel select transistors for each color, turningoff the reset transistors in the array after the interval of time hasexpired, closing the mechanical shutter, wherein the interval of timefor successive rows decreases as a monotonic function. The accumulatedphotocharge may then be read from the pixel sensors for each selectedcolor at a time after but otherwise unrelated to the resetting of thepixel sensors.

The color electronic neutral density filter of the present invention isparticularly suited for imaging arrays having vertical color pixelsensors such as those manufactured by Foveon, Inc., the assignee of thepresent invention, although the present invention is suitable for use inother color imager schemes, such as those employing Bayer patternsensors.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

FIG. 1 is a graph illustrating the optical density of both linear andnon-linear graduated neutral density filters as a function of verticalposition in an image frame.

FIG. 2 is a graph illustrating the density of a linear functiongraduated optical density filter implemented using bladed reset in adigital camera as a function of vertical position in an image frame.

FIG. 3 is a graph illustrating the density of a non-linear graduatedneutral density filter as a function of vertical position in an imageframe.

FIG. 4 is a graph illustrating exemplary control of the 50% point of agraduated optical density filter in accordance with the presentinvention.

FIG. 5 is a graph illustrating exemplary control of the filter densitystrength of a graduated optical density filter in accordance with thepresent invention.

FIG. 6 is a graph illustrating exemplary control of the verticallocation of the lowest filter density strength position of a non-linearneutral density filter having a density that is a non-monotonic functionof vertical (or horizontal) position in an image frame.

FIG. 7 is a graph illustrating exemplary control of the density of agraduated optical density filter having a density that is anon-monotonic function of vertical (or horizontal) frame position inaccordance with the present invention.

FIG. 8 is a graph illustrating exemplary control of the width of acenter minimum-density region of a graduated optical density filterhaving a density that is a non-monotonic function of vertical (orhorizontal) frame position in accordance with the present invention.

FIGS. 9A through 9G are diagrams showing how individual color filterscan be applied in a color imager having separate row reset control overred, green, and blue pixels in individual rows of the imager.

FIG. 10 is a diagram of a digital camera employing a mechanical shutterthat can employ electronic neutral density filters in accordance withthe present invention.

FIG. 11 is a schematic diagram of an illustrative photodiode pixelsensor that can be operated to implement an electronic neutral densityfilters in accordance with the present invention.

FIG. 12 is a timing diagram block diagram illustrating a method ofoperating the photodiode pixel sensor of an image sensor of FIG. 11 toimplement an electronic neutral density filters in accordance with thepresent invention.

FIG. 13 is a block diagram illustrating an exemplary architecture forimplementing an electronic graduated neutral density filter in a digitalcamera in accordance with the present invention.

FIG. 14 is a flow diagram illustrating an exemplary way to control anelectronic graduated neutral density filter in in a digital camera inaccordance with the present invention.

FIG. 15 is a flow diagram illustrating an exemplary way to implement afeedback loop to automatically adjust the filter's position and strengthin accordance with the present invention.

DETAILED DESCRIPTION

Persons of ordinary skill in the art will realize that the followingdescription of the present invention is illustrative only and not in anyway limiting. Other embodiments of the invention will readily suggestthemselves to such skilled persons.

An electronically variable graduated neutral density filter according tothe present invention by controlling the timing of the end of the resetpulse with the closing of a mechanical shutter in the camera, Thistechnique, shown in FIGS. 2 and 3, illustrating optical filter densityas a function of vertical position in the frame may employ either linearattenuation (FIG. 2) or may be non-linear (FIG. 3), for exampleemulating the non-linear density curve of an optical neutral densityfilter. Traces 12 a, 12 b, and 12 c in FIG. 2 show varying degrees oflinear attenuation, and traces 12 d, 12 e, and 12 f in FIG. 3 showvarying degrees of non-linear attenuation.

By applying a non-linear function to a rolling reset (a reset appliedsequentially one row at a time to an imaging array) or a bladed reset inthe image sensor in a digital camera, any non-linear density curve canbe realized. A “bladed reset” is a form of rolling reset where a groupof adjacent rows are simultaneously reset. This group is referred to asa blade. In an image sensor employing bladed reset, the entire imagingarray is reset by resetting the blades one at a time. This is fasterthan resetting the individual rows one at a time using a rolling reset,and is slower than global reset, where all the rows are reset at onetime.

One advantage of a rolling reset and a bladed reset over a global resetis that the artifacts are smaller. The advantage of a bladed reset overa single row rolling reset is the speed is faster. Either a rollingreset or bladed reset can be used for the neutral density filter of thepresent invention. Using the bladed reset method, a trade off is madebetween a larger blade (faster, but larger artifacts) and a smaller one(slower, smaller artifacts). Typical blade sizes for use in the neutraldensity filter of the present invention are 1, 2, 4, 8, 16, 32 rows, butpersons of ordinary skill in the art will readily understand that otherblade sizes may be used depending on the vertical granularity of thefilter artifacts that can be tolerated.

Since the location of the “horizon” that defines the boundary betweendarker foreground & brighter background is a function of the orientationof the camera in the hands of the user and is not fixed, it is useful toallow a user to adjust the vertical position of the filter in additionto its “strength”. This can be done using the 4-way controller buttonslocated on the back of most digital cameras. As non-limiting examples,the position of the 50% attenuation of the electronic filter can beadjusted by using the “up” and “down” buttons 14 on the back of thecamera as illustrated in FIG. 4, a graph showing three exemplary“horizons”, the center one of which is centered on a nominal horizonline (shown at reference numeral 16) in the vertical center of the imageframe.

Similarly, the filter strength (optical density) of the filter can beadjusted by using the “left” and “right” buttons 18 on the back of thecamera as illustrated in FIG. 5, a graph showing three differentexemplary filter gradient curves 20 a, 20 b, and 20 c for a singlehorizon setting. Persons of ordinary skill in the art will easily beable to implement such control functions in a digital camera using theseor other control buttons or surfaces on the camera.

The filter adjustments can be seen in an electronic viewfinder (LCDscreen) of the digital camera, allowing the user to preview the effectsand make fine adjustments if desired.

In accordance with the present invention, an optical density filterhaving a density that is a non-monotonic function of vertical (orhorizontal) position in an image frame may be realized. FIG. 6 is agraph illustrating exemplary control of the vertical location of thelowest filter density strength position of a non-linear neutral densityfilter having a density that is a non-monotonic function of vertical (orhorizontal) position in an image frame.

A filter having the characteristics shown in FIG. 6 may be useful tocapture images where the composition includes, for example, a bottomground portion, a central sky portion having a brightness higher thanthe bottom ground portion, and an upper cloud portion having abrightness lower than the central sky portion, or a compositionincluding a patch of sky between high-rise buildings. Other exampleswill readily suggest themselves to photographers.

The vertical portion of the image where the lowest density portion ofthe filter is centered may be adjusted using the 4-way controllerbuttons located on the back of most digital cameras. As non-limitingexamples, the vertical position of the lowest density portion of theelectronic filter can be adjusted by using the “up” and “down” buttons14 on the back of the camera as illustrated at reference numerals 22 a,22 b, and 22 c in FIG. 6, that shows the vertical position of threeexemplary “horizons” a, b, and c, the center one of which (b) iscentered on a nominal horizon line (shown at reference numeral 16) inthe vertical center of the image frame.

Similarly, the filter strength (optical density) of the filter can beadjusted by using the “left” and “right” buttons 18 on the back of thecamera as illustrated in FIG. 7, a graph showing three differentexemplary filter gradient curves 24 a, 24 b, and 24 c for a singlehorizon setting. Persons of ordinary skill in the art will easily beable to implement such control functions in a digital camera using theseor other control buttons or surfaces on the camera. The width of thelowest density portion of the filter can also be adjusted as shown inthe graph of FIG. 8, where the widest setting is represented by filtergradient curve 26 a, a wider filter gradient curve 26 b, and the widestfilter gradient curve 26 c. The function button(s) on a digital cameracan be used to define the functions of the up, down left and rightbuttons as is well known in the art.

Persons of ordinary skill in the art will recognize that filtersimplementing other non-monotonic functions may be realized according tothe present invention.

With respect to digital camera image sensors that implement the resetfunction as a row reset function, persons of ordinary skill in the artwill appreciate that there is a fundamental limitation to this techniquein that the filter only works when the camera is held in “landscape”mode. This covers most landscape photos, and fits well with the mainapplication for high-resolution digital cameras. Image sensors thatindependently control row and column reset functions can be configuredto allow operation in either portrait or landscape mode.

According to another aspect of the present invention, an independentcolor channel graduated density filter allows the optical density ofindividual color pixel sensors in an imaging array to be controlledindependently. The present invention is particularly suited for imagingarrays having vertical color pixel sensors such as those manufactured byFoveon, Inc., the assignee of the present invention, although theoperation of the present invention is not limited to such sensors.

The ability to independently control the optical density filters in theseparate colors of a multi color sensor with the horizon set at anarbitrary position, such as, but not limited to, the bottom of the imageframe allows different artistic effects to be implemented.

Since the shutter speed varies, it is particularly advantageous to usethe maximum exposure time to calculate it. Then the filter strength maybe adjusted in accordance with the principles of the present invention.

Since the shutter speed varies across the frame in image sensorsemploying the present invention, motion artifacts may occur if theexposure time is short and high filter strength is used. In typicallandscape shots this should not cause a problem. In fact, it can bebeneficial. One of the difficult challenges is to properly expose forthe foreground when there is motion in the sky, such as birds flying orclouds moving rapidly. In high dynamic range (HDR) photography, donewith exposure bracketing, this results in streaking, double images, etc.The electronic neutral density filter of the present invention will helpto freeze motion in such cases and will also help minimize the effect ofartifacts such as leaves moving in the breeze, another difficult casefor HDR techniques. In addition, persons of ordinary skill in the artwill appreciate that such motion artifacts may be deliberately exploitedfor producing various artistic effects.

According to one aspect of the present invention illustrated in FIGS. 9Athrough 9G, by using the electronic density filter techniqueindependently for different color channels in accordance with thepresent invention, graduated filters of any color can be created. FIG.9A shows an example of a full-spectrum (white) filter; FIG. 9B shows anexample of a red filter; FIG. 9C shows an example of a green filter;FIG. 9D shows an example of a blue filter; FIG. 9E shows an example of amagenta filter; FIG. 9F shows an example of a cyan filter; and FIG. 9Gshows an example of a yellow filter.

As seen in FIGS. 9A through 9G, the filter can be graded between neutral(by controlling the three colors by an equal amount) and any color orcombination of colors. Persons of ordinary skill in the art will readilyappreciate that the examples shown in FIGS. 9A through 9G areillustrative only and that the exposure time of each of the individualRGB filters is separately and potentially infinitely variable betweenzero and a maximum exposure time.

The independent color channel density filter of the present inventioncan also be used to create gradation in hue across the electronicfilter. As a non-limiting example, in black and white photography thesky above the horizon line could be darkened using yellow filtrationwhile blue flowers below the horizon line are lightened with bluefiltration.

It is possible to make such filters for sensors with a color filterarray, such as a Bayer pattern filter. In a Bayer pattern filter, eachrow of the array needs to have two row reset control lines. Alternaterows of the array need to have separate control over reset of green andred pixels and green and blue pixels.

As long as only neutral density filters are used, vertical color sensorssuch as the X3 sensor designed by Foveon of Santa Clara, Calif. does notoffer any advantages over CFA sensors.

When colored filters are used in imagers having separate control overrow reset of red, green, and blue, pixels, CFA sensors will suffer aloss of resolution.

Vertical color sensors such as the X3 sensor require a minimum area tocreate three independent R, G, B values. Sensors such as the X3 sensorrequire only 1/9^(th) the area required by a Bayer pattern CFA sensor,since all colors are arranged vertically and are independentlycontrollable. In addition vertical color sensors do not suffer any lossof resolution, and will therefore produce higher-quality images.

Referring now to FIG. 10, a diagram shows an illustrative digital camera30 in which the present invention may be implemented. A scene 32 isfocused by lens 34 through mechanical shutter 36 onto imaging array 38.Control circuits 40 are usually disposed on the imaging array itself.Control circuits 40 include exposure and readout controls for operatingthe rows and columns of the pixels in the array as is known in the art.A user may interact with and control camera 30 through user interface42.

Referring now to FIG. 11 and FIG. 12, an aspect of the invention will bedescribed with reference to a three-color pixel sensor. Persons ofordinary skill in the art will appreciate that the operation of amonochromatic pixel sensor in accordance with the present invention isunderstood by considering a single one of a plurality of possiblecolors. FIG. 11 is a schematic diagram of an illustrative three-colorpixel sensor 50 that can be used to implement the present invention.Exemplary colors red, green, and blue are used to disclose this aspectof the invention but persons of ordinary skill in the art willappreciate that the present invention is not limited to the use of theseparticular colors. Each pixel sensor in an array of pixel sensors isdisposed at a row position and a column position in the array. Pixelsensor 50 is one of a plurality of pixel sensors disposed in a row ofthe array at a particular column position.

Red photodiode 52 is coupled through red-select transistor 54 to afloating node 56. Red-select transistor 54 has its gate driven byred-select line 58. There is one red-select line for every row in thearray.

Green photodiode 60 is coupled through green-select transistor 62 to thefloating node 56. Green-select transistor 62 has its gate driven bygreen-select line 64. There is one green-select line for every row inthe array.

Blue photodiode 66 is coupled through blue-select transistor 68 to thefloating node 56. Blue-select transistor 68 has its gate driven byblue-select line 70. There is one blue-select line for every row in thearray.

Floating node 56 is coupled to a reset potential through resettransistor 72. The reset transistor 72 has its gate driven by row-resetline 70. There is one row-reset line for every row in the array.

The floating node 56 is coupled through a source-follower transistor 76and a row-select transistor 78 to a column line 80 for reading out thecharge accumulated by the photodiodes 52, 60, and 66. The row-selecttransistor 78 has its gate driven by a row-enable line 82. The columnline 80 may be driven by a current source 84 as is known in the art. Thecolumn line 80 is discharged to a column discharge potential at selectedtimes by applying a column discharge potential to the gate of columndischarge source-follower transistor 86.

FIG. 12 is a timing diagram that illustrates the timing signals appliedto the pixel sensor 50 to implement the electronic neutral densityfilter in accordance to one aspect of the present invention. FIG. 12shows the various signals that can be applied to the control lines ofthe pixel sensor 50 to implement a method in accordance with the presentinvention.

Exposing an image is controlled in part by the mechanical shutter of thecamera (reference numeral 36 in FIG. 10). The opening and closing of themechanical shutter is depicted by the first trace 90 in the timingdiagram of FIG. 12.

The second trace 92 of FIG. 12 represents the signal applied to therow-reset line 74 of FIG. 11. In the normal operation of a pixel sensorof the type depicted in FIG. 11, each pixel is reset by simultaneouslyturning on its associated select transistor and the reset transistor.After the associated select transistor has been turned off, charge isallowed to accumulate on the photodiode until the mechanical shuttercloses, blocking any additional light from reaching the imaging array.The accumulated charge is read out after discharging the column line ata time dependent on the length of the exposure required for the ambientlight level of the subject. As will be appreciated by persons ofordinary skill in the art, the dark level of the pixel sensor will alsobe read out at some point during the read out process. In normaloperation, the timing of signals to reset and read each pixel sensor 50is selected to assure that each color is exposed for the same amount oftime.

In accordance with the present invention, the timing of the controlsignals applied to the pixel sensor 50 is altered to implement theelectronic neutral density filter in accordance with the presentinvention. In the illustrative example of FIG. 9, the timing of thesignals shown is selected to produce an integration time t₁ for the redphotodiode 52, an integration time t₂ for the green photodiode 60, andan integration time t₃ for the blue photodiode 66, where t₁>t₂>t₃.Persons of ordinary skill in the art will appreciate that because thereset, red-select, green-select, and blue-select signals are row-wise orblade-wise signals, all pixel sensors in a row (or group of adjacentrows constituting a blade), the operations described herein will occurin all pixel sensors 50 in the selected row or blade.

As may be seen from an examination of FIG. 12, the row-reset signal 92is asserted prior to the opening of the mechanical shutter 90. Thered-select transistor 54 is turned on by the red-select signal 94asserted on red-select line 58. When both the red-select transistor 54and the reset transistor 72 are turned on, the red photodiode 52 in thepixel sensor 50 is held at a reset potential.

The red-select signal 94 is de-asserted at a time selected to produce anintegration time t₁ for the red photodiode 52 in the pixel sensor 50.The integration time t₁ ends when the mechanical shutter closes,blocking any additional light from reaching the pixel sensor 50 in theimaging array.

The green-select transistor 62 is turned on by the green-select signal96 asserted on green-select line 64. When both the green-selecttransistor 54 and the reset transistor 72 are turned on, the greenphotodiode 60 in the pixel sensor 50 is held at the reset potential.

The green-select signal 96 is de-asserted at a time selected to producean integration time t₂ for the green photodiode 60 in the pixel sensor50. The integration time t₁ ends when the mechanical shutter closes,blocking any additional light from reaching the pixel sensor 50 in theimaging array.

The blue-select transistor 68 is turned on by the blue-select signal 98asserted on blue-select line 70. When both the blue-select transistor 68and the reset transistor 72 are turned on, the blue photodiode 66 in thepixel sensor 50 is held at a reset potential.

The blue-select signal 98 is de-asserted at a time selected to producean integration time t₃ for the blue photodiode 66 in the pixel sensor50. The integration time t₃ ends when the mechanical shutter closes,blocking any additional light from reaching the pixel sensor 50 in theimaging array.

The closing of the mechanical shutter stops the integration ofphotocharge on the red, green, and blue photodiodes 52, 60, and 66. Thisallows the readout of the accumulated photocharge from the red, green,and blue photodiodes 52, 60, and 66 to be completely decoupled from thereset timing of the red, green, and blue photodiodes 52, 60, and 66 inpixel sensor 50. Readout is shown using the different color selectsignals 94, 96, and 98, while row select signal 100 is high.

The columns are discharged before each color read as shown at referencenumeral 102.

Persons of ordinary skill in the art will observe that it is notrequired for the row-reset signal 92 to be on during the entire timethat the mechanical shutter is held open. It is only necessary to assertthe row-reset signal during the times that the red-select, green-select,and blue-select signals 94, 96, and 98 are asserted while the mechanicalshutter is open. Such skilled persons will also appreciate that, asshown in FIG. 12, the longest integration time t₁ that starts at thefalling edge of the red-select signal 94 can be set to commence afterthe mechanical shutter has completely opened, eliminating any timingvariance in the mechanical shutter opening from affecting theintegration time. Persons skilled in the art will appreciate that thisparticular mode of operation may be limited at extremely fast exposuretimes.

As previously noted, one advantageous aspect of the present invention isthat the readout of the accumulated photocharge from the red, green, andblue photodiodes 52, 60, and 66 is completely decoupled from the resettiming of the red, green, and blue photodiodes 52, 60, and 66 in pixelsensor 50, subject only to limitations imposed by any parasitic chargeleakage. For the particular pixel sensor 50 depicted in FIG. 11, readoutof each color is accomplished by first discharging the column line byasserting a signal to the gate of column discharge transistor 86 andthen asserting one of the color select signals 94, 96, or 98.

Referring now to FIG. 13, a block diagram illustrates an exemplaryarchitecture for implementing an electronic graduated neutral densityfilter in a digital camera in accordance with the present invention. Theimaging circuitry 110 includes an imaging array 112 containing an arrayof rows and columns of pixel sensors such as the pixel sensor depictedin FIG. 11. Imaging array 112 is controlled by array control circuits114 that supply the signals to drive row-reset lines 116, row-enablelines 118, color-select lines 120, and column-reset lines 122. Personsof ordinary skill in the art will appreciate that in imaging arrayswhere the source-follower transistors are not shared, the color selectlines 120 are not necessary.

A user interface 124 accepts input from a user to control the camera asis well understood in the art. In addition to customary user input, theuser can control operation of the electronic neutral density filter ofthe present invention. User commands are sent from the user interface124 to the array control circuits 114 and to logic 126. The function oflogic 126 is to direct the operation of the array control circuits 114as is known in the art and to also direct the operation of theelectronic neutral density filter by writing the appropriate data tofilter data tables 128 to control the timing of the signals drivingrow-reset lines 116, row-enable lines 118, color-select lines 120, andcolumn-reset lines 122. The filter data tables 128 implement theuser-selected filter strength and horizon location selections shown, forexample, in FIGS. 4 and 5. Persons of ordinary skill in the art willreadily observe that the number of filter data tables 128 will correlatewith the number of colors to be implemented in the electronic neutraldensity filter being implemented. Such skilled persons will alsoappreciate that mathematical operations such as polynomial or otherarithmetic functions may be used to implement the same function as thetables.

Referring now to FIG. 14, a flow diagram illustrates an exemplary method130 that may be used to control an electronic graduated neutral densityfilter in in a digital camera in accordance with the present invention.

The method starts at reference numeral 132. At reference numeral 134 thecamera enters neutral density filter mode, typically by selecting thatmode from the choices a camera menu. Implementation of camera menus iswell known in the art. At reference numeral 136, it is determinedwhether the up or down button of the camera is activated. If the upbutton is activated, the process proceeds to reference numeral 138,where the horizon position is moved up. If the down button is activated,the process proceeds to reference numeral 140, where the horizonposition is moved down. If neither the up nor down buttons areactivated, the process proceeds to reference numeral 142, where it isdetermined whether the left or right button of the camera is activated.If the left button is activated, the process proceeds to referencenumeral 144, where the filter strength is decreased.

If the right button is activated, the process proceeds to referencenumeral 146, where the filter strength is increased. Persons of ordinaryskill in the art will appreciate that the order of sensing the up/downbuttons and left/right buttons is arbitrary and either pair of buttonscan be sensed before the other. Such skilled persons will alsoappreciate that the assignment of the button functions is also somewhatarbitrary and that the exact response of the method to the user inputwill depend on the action desired, such as incremental or scrollingactions.

If neither the left or right buttons are activated, the process proceedsto reference numeral 148, where it is determined if a command to exitthe filter mode has been asserted. If the command to exit the filtermode has been asserted, the process ends at reference numeral 150. Ifthe command to exit the filter mode has not been asserted, the processreturns to reference numeral 136 and the polling of the up/down andleft/right buttons resumes.

In accordance with other aspects of the present invention, it is alsopossible, with several live view frames, to implement a feedback loop toautomatically adjust the filter's position and strength. Implementationof such a feedback loop will provide a good default filter; the user canthen “tweak” it from that position if needed. Implementation of anexemplary one of such feedback loops is show in FIG. 15.

Referring now to FIG. 15, a flow diagram illustrates an exemplary method160 for implementing a feedback loop to automatically adjust thefilter's position and strength in accordance with the present invention.The method begins at reference numeral 162.

At reference numeral 164 an initial frame (e.g., a live view frame) isexposed and an initial frame histogram is generated. At referencenumeral 166 it is determined whether the shadow signal-to-noise ratio issufficient. This ratio will vary in individual cases as is known in theart and can be selected for an individual camera model.

If the shadow signal-to-noise ratio is not sufficient, the processproceeds to reference numeral 168 where the global exposure of thecamera is increased. At reference numeral 170 a new frame is exposed andnew histogram is generated from the new frame. The process then returnsto reference numeral 166 where it is it is determined whether the shadowsignal-to-noise ratio is sufficient. This loop is repeated until it isdetermined that the shadow signal-to-noise ratio is sufficient.

If at reference numeral 166, it was determined that the shadowsignal-to-noise ratio is sufficient, the process proceeds to referencenumeral 172, where a graduated neutral density filter having a nominal50% horizon position and a trial density level is applied. At referencenumeral 174 a new frame is exposed and a new histogram is generated fromthe new frame. At reference numeral 176 it is determined whether anyhighlights of the image are clipped. If any highlights of the image areclipped, the process proceeds to reference numeral 178, where the filterdensity is increased. The process then returns to reference numeral 174where a new frame is exposed and a new histogram is generated from thenew frame. At reference numeral 176 it is again determined whether anyhighlights of the image are clipped. This loop is repeated until it isdetermined that no highlights of the image are clipped.

If it is determined at reference numeral 176 that no image highlightsare clipped, the process proceeds to reference numeral 180 where thehistogram spread of the current image is measured. Next, at referencenumeral 182, the horizon point of the filter is raised and at referencenumeral 184 a new frame is exposed and a new histogram is generated fromthe new frame. At reference numeral 186 the histogram spread of thecurrent image exposed at reference numeral 184 is compared with thehistogram spread of the previous image that was determined at referencenumeral 180. If the histogram spread of the current image exposed atreference numeral 184 is lower than the histogram spread of the previousimage that was determined at reference numeral 180, the process returnsto reference numeral 182, where the horizon point of the filter israised. The method loops through reference numerals 182, 184, and 186until the histogram spread of the current image is higher than that ofthe previous image. At that point the process proceeds to referencenumeral 188, where the previous horizon level is restored.

The process then proceeds to reference numeral 190, where the horizonpoint of the filter is lowered. At reference numeral 192 a new frame isexposed and a new histogram is generated from the new frame. Atreference numeral 194 the histogram spread of the current image exposedat reference numeral 192 is compared with the histogram spread of theprevious image. If the histogram spread of the current image exposed atreference numeral 184 is lower than the histogram spread of the previousimage that was determined at reference numeral 180, the process returnsto reference numeral 190, where the horizon point of the filter islowered. The method loops through reference numerals 190, 192, and 194until the histogram spread of the current image is higher than that ofthe image exposed using the horizon of reference numeral 188. At thatpoint the process proceeds to reference numeral 196, where the previoushorizon level is restored. The method ends at reference numeral 198.

Persons of ordinary skill in the art will appreciate that the shiftingof the horizon point may alter the shadow signal-to-noise ratio, whichmay be re-calibrated at this point as shown at reference numerals 166,168, and 170. The image may also be re-examined and corrections made forany clipped highlights as shown in the loop including reference numerals176 and 178 and 174.

Persons of ordinary skill in the art will observe that the techniques ofthe present invention can be used to make a “flat” exposure (an exposuremade with the camera fstop range set to 0, making the exposure the sameacross the entire imager). Flat exposures may be created by programmingthe reset turnoff to lead the shutter turnoff by the same time at eachrow as the shutter blade turnoff crosses the imager. Persons of ordinaryskill in the art will appreciate that a typical focal plane shutterchanges speed considerably across the imager, so it is necessary togenerate a nonlinear curve of reset travel characteristic of the shutterin order to provide the shutter turnoff time for each row. Using thistechnique may allow cheaper or shorter exposure focal plane shutters tobe used.

While embodiments and applications of this invention have been shown anddescribed, it would be apparent to those skilled in the art that manymore modifications than mentioned above are possible without departingfrom the inventive concepts herein. The invention, therefore, is not tobe restricted except in the spirit of the appended claims.

What is claimed is:
 1. In a digital camera having an imaging arrayformed in a semiconductor body and including a plurality of pixelsarranged in rows and columns, each pixel including first, second, andthird color sensors aligned with each other at different depths in thesemiconductor body, the digital camera further having a mechanicalshutter having an open position allowing light to illuminate the imagingarray and a closed position blocking light from illuminating the imagingarray, a method for performing color density filtering of imagescaptured by the imaging array, the method comprising: opening themechanical shutter; operating each row in the array by: resetting all ofthe color sensors in each pixel in the row by asserting a row resetsignal common to all color sensors in all pixels in the row and then,for the first, second, and third color sensors in each pixel in the rowseparately asserting first, second, and third color-select signals forthe first, second, and third color sensors, respectively, of all of thepixel sensors in the row at times after the mechanical shutter isopened; starting exposure for the first color sensor in each pixel inthe row at a first color sensor exposure start time by then de-assertingthe first-color select signal for all of the first color sensors for allpixels in the row, the exposure start time for the first color sensor ineach row in the array being a function of a density filter function forthe first color applied to an image to be captured by the array;starting exposure for the second color sensor in each pixel in the rowat a second color sensor exposure start time by then de-asserting thesecond-color select signal for all of the second color sensors for allpixels in the row, the exposure start time for the second color sensorin each row in the array being a function of a density filter functionfor the second color applied to an image to be captured by the array;starting exposure for the third color sensor in each pixel in the row ata third color sensor exposure start time by then de-asserting thethird-color select signal for all of the third color sensors for allpixels in the row, the exposure start time for the third color sensor ineach row in the array being a function of a density filter function forthe third color applied to an image to be captured by the array; endingthe exposure for the first, second, and third color sensors for all ofthe pixel sensors in the row by closing the mechanical shutter; andreading color exposure values from the first, second, and third colorsensors in each row of the pixel array by separately asserting thefirst, second, and third color-select signals for all of the pixelsensors in the row at first, second, and third color read times afterthe mechanical shutter has closed, the first, second, and third colorread times being independent of one another and being unrelated to thestart times for first, second, and third color sensors; wherein thefirst, second, and third color exposure start times are independent ofone another.
 2. The method of claim 1, wherein the first, second, andthird color exposure start times, respectively, are the same for n rowsin the array, where n is an integer greater than
 1. 3. The method ofclaim 1, wherein the first, second, and third color exposure starttimes, respectively, are the same for n successive rows in the array,where n is an integer greater than
 1. 4. The method of claim 1 whereineach row in the array is operated sequentially from a first row in thearray to a last row in the array.
 5. The method of claim 1 wherein thedensity-filter function of each of the first, second, and third colorsis a monotonic function.
 6. The method of claim 1 wherein thedensity-filter function of each of the first, second, and third colorsis a non-monotonic function.
 7. The method of claim 1 wherein the rowreset signal is asserted before opening the mechanical shutter.
 8. Themethod of claim 1 wherein: asserting the row reset signal places a resetpotential on a common node in each pixel; and separately asserting thefirst, second, and third color-select signals for the first, second, andthird color sensors couples the common node in a pixel respectively tothe first, second, and third color sensors in the pixel.
 9. The methodof claim 1 wherein: the first color sensor in each pixel is a redsensor; the second color sensor in each pixel is a green sensor and ablue sensor; and the third color sensor in each pixel is a blue sensor.